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ABSTRACT 


The purpose of this thesis is a discussion of developing human-like 
behavior in the computer. A theory of the human learning processes is 
first described. This leads to the presentation of a computer game 
which simulates the human capabilities of reasoning and learning. The 
program is required to make intelligent decisions based on past experi- 


ences and critical analysis of the present situation. 
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I. INTRODUCTION 


In the past few years, much attention in the computer world has 

been given to the study and development of Artificial Intelligence. 

One goal of artificial intelligence is to develop human-like qualities 

in the computer. One method of patterning human behavior is achieved by 
giving the computer the ability to (1) absorb data, (2) use inductive 
reasoning to make generalizations based on the data and abstract informa- 
tion from the data, (3) make decisions based on these abstractions and 
generalizations, and (4) learn from past experience. Admittedly this 
seems to be an enormous task, and in fact it is, yet this is how a child 
learns. To go one step ante in modeling human behavior, and at the 
same time provide a reasonable limit on the size of the computer required, 
the ability to "forget" long-past experience could be developed in the 
system. 

The theory of the human learning process is discussed first in this 
thesis. The theory leads to the development of a computer program that 
learns based on data gathered from past experiences. The program uses 
its library of stored knowledge in making decisions. The example which 
is presented is a game named DIPLOMAT which simulates the representatives 
of two nations interacting in strategic negotiations regarding relative 
strengths, i.e., armaments. The game is played between two opponents, 
the computer and a person. 

The authors of the thesis consider that the interaction of the com- 
puter and a human in a decision-making environment is an excellent way 
of demonstrating the ability of the computer to develop human-like be- 


havior. 





Il, QHEORY 


One of the objectives of artificial intelligence is to build a 
computer system that will effectively mimic the human learning process. 
This would be an enormous task if the system were to be so general in 
nature that it could handle any conceivable task, because it would in- 
volve representing the external world in a fonn adaptable for the com- 
puter. When restricted to a small subset of the world, i.e., confined 
to one or two specific tasks, attempts at this objective have been fairly 
successful in that the systems do mimic and sometimes surpass the human 
in their performance in these areas. However when trying to model the 
human brain in the general sense, the problem has been found to be ex- 
tremely complex. 

A possible approach to the problem would be to model the human brain 
using the physiological approach, i.e., to build the exact electrical 
network of sensors, storages, <nd connectors that form the physical makeup 
of the brain. Obviously this is impractical because the brain is com- 
posed of so many cells. Dr. R. L. Beurle, a noted English authority on 
artificial intelligence [Ref. 3] who has extensively studied the theory 
of brain models, estimates that the brain is composed of approximately 
ia neurons (nerve cells). 

Another approach to the problem might be the psychological ap- 
proach. In this approach, it is necessary to model the logical structure 
of the brain rather than the physical structure. However in order to use 
this approach, some knowledge of the process of psychological development 
of human behavior must be obtained. It would be best to trace the learn- 


ing processes from their beginning in a child, since a knowledge of 
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total human behavior usually requires almost a lifetime of study and 
experience. 

In the area of the development of the mental abilities of the child, 
the works of one man stand out above all others - the works of Jean Piaget. 
Jean Piaget is a noted Swiss child psychologist who was educated at 
Hevchotel, Zurich, and the University of Paris. He has been a professor 
of child psychology and history of scientific thought at the University 
of Geneva since 1929, and is the director of the International Bureau of 
Education at the Institute J. J. Rousseau. Above all, he is noted for 
his research in the development of the mind from birth to adolescence. 

The works of Jean Piaget, as presented in References 9 through ll, 
form the basis of this ee on the theory of the development of the 
human learning process. They are supported by Refecences 15 and 16, 


which are basic references used in the American medical profession. 


A. THE LEARNING PROCESS OF THE HUMAN MIND 

Piaget contends that the human mind consists of a finite number of 
structures, each consisting of a finite number of cells. Each of these 
cells contains an element of information which is a part of the human 
thought process. He theorizes that there is a separate set of informa- 
tion structures for each function of the central Peon system which is 
composed of the brain and spinal cord. Many of these separate sets of 
Structures are developed before birth, for example, a set controlling 
the action of the heart, another the function of breathing, a third 
controlling the flexing of the arms and legs. Piaget further contends 
that as relationships build, these structures are linked together to 


produce automatic reflexes and in general the structures are reordered 
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and enlarged in successive phases. The central theme for the human 
learning process is then the process of manipulation of the infornation 
structures, i.e., the building, storing, linking, rebuilding and re- 
linking of elements of information. 

It is easiest to describe the manipulating of these information 
structures by studying the development of the brain of the newborn baby. 
This is the approach presented by Piaget. At birth the cerebral cortex, 
which is composed principally of neurons, is largely undeveloped, and 
the infant is basically a reflex organism. The reflexes are the result 
of the manipulation of the information structures before birth. Other 
than the regular actions of the heart and other vital systems, most muscle 
activity is random, lacking direction from the brain. 

One of the first learning processes of the infant is the learning 
of spatial relations’.ips. While flexing his «rms and legs, he touches 
the side of the crib. Relationships are built in his mind between the 
information structures for the sense of touch and those of muscle control, 
and a new structure built up regarding an awareness of the confining walls 
of the crib. These structures become libraries of information. Other 
libraries build up, for example he grows to associate his mother with 
warmth, comfort, and nourishment. The building of relationships between 
the structures, and the reordering of the structures and links, forms an 
associative memory within the brain. 

It is important in the study of the human brain to understand that 
these activities are the result of relationships between the libraries 
even though the information in their structures is dissimilar. Once the 
links are established, the dissimilar sets function together. The act of 


crying which is the result of combining the structures of muscle control, 
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an awareness of a specific need, respiration control, and others is a good 
example. The operation that is accomplished constitutes some action of 
uniting or separating, placing or displacing, arranging or disarranging 
elements of information into sets of structures. As the relationships 
develop within the brain, the mind learns how to use the data which has 
been structured within its memory to produce a desired result, such as 


rolling over or sitting up. 


Be REPRESENTATION OF THE HUMAN LEARNING PROCESS 

The process of thought is the result of the human becoming aware of 
the relationships that link cell structures in the associative memory of 
his brain. Piaget explains the mechanism of thought as a movement which 
evolves when an Beeren cee of the relationships becomes sufficiently 
advanced to permit the individual to combine the information from several 
structures into a single idea. The resulting thought may cause other re- 
actions such as body movement, and thus may cause new structures to be 
created or new links to be built. The actual process involved in the 
human brain in conceiving a thought is not definitely known, and must be 
extremely complicated. If such a scheme is to be computerized, however, 
it must be made deterministic. 

In an attempt to make the process a deterministic one, it is neces- 
Sary to summarize some facts deduced from the discussion of Piaget's 
works. One result of the thought process is the acquiring of an idea, 
which is the outcome of the interaction of cells or structures. The 
combination of old structures in a new way may lead to new and possibly 
improved concepts. The new relationships may be formed by applying de- 
ductive reasoning to some information extracted from the interconnected 


structures. In this way, the learning of a new concept may be achieved. 
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Thus learning is not merely an additive process, i.e., the piling of 
one disjoint piece of information atop another and another and another. 

The number of cells or structures is not the criteria for learning, 
rather it is the effective combining of the stored information that re- 
sults in intelligence. Once a certain point is reached in the process 
of human development, the physical size of the brain does not rapidly 
become larger and larger, rather information is restructured, old informa- 
tion may be forgotten and new information stored in its place, and new 
links established. 

These two techniques, structuring of memory and learning by inter- 
action of the structured information, are utilized in the structuring of 
a computer memory and dhe Qsnelorngae of a program which wi!l exhibit de- 
ductive reasoning based on learning. This is the example presented in 
this thesis. Considerable research and soul searching went into formu- 
lating a worthwhile application for the model of human intelligence. 


The application had to be general enough to apply to real-life situa- 


tions, yet not overly complicated. 


C. BACKGROUND OF THE GAME OF DIPLOMAT 

Several recent periodicals have been devoting considerable attention 
to games as an application of artificial intelligence. One game in parti- 
cular has been the subject of much of the attention, the game known as 
the Prisoner's Dilemma. The classic prisoner's dilemma is described in 
reference 7 as follows: 

"Two suspects are taken into custody and separated. The 
district attorney is certain that they are guilty of a specific 
crime, but he does not have adequate evidence to convict them 
at a trial. He points out to each prisoner that each has two 


alternatives: to confess to the crime the police are sure they 
have done, or not to confess. If they both do not confess, then 
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the district attorney states he will book them on some minor 

trumped-up charge such as petty larcency and illegal possession 

of a weapon, and they will both receive minor punishment; if 

they both confess they will be prosecuted, but he will recom- 

mend less than the most severe sentence; but if one confesses 

and the other docs not, then the confessor will receive lenient 

treatment for turning state's evidence, whereas the latter will 

get 'the book' slapped at him." 
One reason the prisoner's dilemma has been discussed so much is that there 
are numerous situations in the world that have some of the characteristics 
of this game (or extensions from this game such as the addition of parti- 
cipants and/or strategies). Most economic situations that require a 
choice among a finite number of strategies have these characteristics. 
Consider, for example, gasoline service stations located close to one an- 
other, each of which can lower its prices. Regardless of the price one's 
competitors set, any one manager is better off, in the short run at least, 
cutting his price. If all cut prices, however, the total volume of bus- 
iness is the same as if none cut prices, but the total revenue is less. 
On a larger scale, consider wheat farmers in a country without govern- 
mental price and production controls. Any one farmer is better off pro- 
ducing wheat as long as his marginal cost is not greater than the price. 
He will be able to sell all he can produce at the going market rate with- 
out affecting the price. If all farmers produce maximum amounts, however, 
the price will be pushed down and all will be worse off than if each had 
restricted his production. On the worldwide scale, there is the problem 
of disarmament. One country can be more powerful (or secure) by arming, 
but nothing is gained if all arm. All countries would be better off if 
all disarmed in that the money not spent on defnese could be spent for, 
say, consumer goods or for correcting social ills. 


More interesting than the one-time classic prisoner's dilemma is the 


iterated game, i.e., a game composed of many moves. In the overall 
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picture of the iterated game, each player must (in general) forsake the 
possibility of maximizing his own short-run profit to enjoy the greatest 
payoff by maximizing his long-run profit. With a one-trial game and an 
unknown rival, it is difficult to imagine the wisdom of choosing a medium- 
gain, little-risk, cooperating type strategy, when more can be gained (or 
lost) by choosing the high-risk high-payoff strategy (nothing ventured, 
nothing gained). The single trial Peas cin eliminates both the possibil- 
ity of future cooperation and the possibility of punishing a rival for non- 
cooperative action in one trial. Dr. Lester B. Lave of the Carnegie 
Institute of Technology [Ref. 6] has studied factors affecting cooperation 
in prisoner's dilemma type games. He has found that the single-trial game 
and the multi-trial game are basically equivalent in the formal sense in 
that the expected values of the two games have the same range acress differ- 
ent groups of opponents. However, the games are quite different with re- 
Spect to negotiating cooperation among different participants. The ex- 
pected values of the two games are not equal for a given rival, since 
certain forms of behavior can induce cooperation or competition. He based 
these results on experiments conducted using human competitors only, and 
did not introduce computer gaming into his research. He further found 
through experimentation that when a game was iterated, it was possible 
to display behavior that induces or stifles cooperation. He also found 
that it was possible for the players to develop communication between 
them usin~ the choices in the game, but that this rudimentary form of 
communication took time to establish and function. He found that the 
longer the game, the more likely it was that a stable cooperative solu- 
tion could be achieved. 

Other experiments with a complex decision task showed that experi- 


ence fron previous tasks was a large factor in success. The conclusions 
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drawn from studies conducted at the Human Performance Center, Ohio State 
University, [Ref. 13] were that after gaining experience when tested under 
realistic circumstances, the experienced subjects were in general less 
conservative than naive subjects who received no such prior training. 

They were willing to take more risks to achieve Rieuee overall gains. 

The study of concept attainment by the machine has also been the 
subject of study. In order for a human to may a game of this nature, 
he must be able to form a few concepts of the game itself and of his 
opponent. It can be likened to the game of poker in that opponents must 
deduce the type of individuals playing the game. Dr. Frank B. Baker, a 
professor of educational psychology at the University of Wisconsin, has 
studied the theory of concept attainment and developed a computer program 
which demonstrated the theory in a simple decision task requiring the 
identification of common attributes among different sets [Ref. 1]. In 
Reference 2, Dr. Baker states: | 

"If computer programs are to serve as useful models of 
congnitive behavior, their creators cannot avoid coming to 

grips with the necessity for establishing an internal organi- 

zation for their model which implements the higher level 

cognitive behavior associated with the human capacity for 

self-direction, autocriticism, and adaptation." 

In computer game playing, the concept of the game itself is aul into 
the game, however the concept of different methods of play or types of 
strategies that the computer may face is something that must be attained 
as it plays. 

Most of the behavioral tests conducted to date have been between 
humans. However, in the past few years, more attention has been devoted 
to simulating these tests on the computer using interaction between man 
and the machine. Professor Roman J. Weil of the University of Chicago 


stated in Reference 18 the advantage of using the computer for this appli- 


cation quite well: 
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"The philosophy underlying the computer approach is this: 

If a program can be constructed that, when placed in a prisoner's 

dilemma situation, exhibits behavior like the behavior of people 

when placed in the same situation, then that program will be a 

powerful tool for generalizations." 

Professor Weil goes on to say that if the computer can be made to 
simulate the human in organizing data and making decisions under all 
simulated conditions of risk and stress, it will be possible to accumu- 
late more data and more accurately predict human behavior in the same 
environment. 

The game of DIPLOMAT presented in this thesis is basically an ex- 
tended version of the prisoner's dilemma. It incorporates an iterated 
game technique with the game lasting anywhere from ten to fifty moves, 
and two opponents choosing from among three strategies. Additional compli- 
cations to the prisoner's dilemma basis are inserted by varying the pay- 
offs to the participents as a function of previous moves, and by inserting 
an unknown random variance into the payoff table. 

In order to be successful in this game, the participants must per- 
form most of the tasks listed in the introduction as a goal of artifical 
intelligence, in addition to performing the task of concept attainment. 

In particular, the computer must analyze the situation of the game at 

the time of the move and refer to past games and past moves in the pre- 
sent game to determine its opponent's probable strategy. It must then 
abstract enough information from his prior and present knowledge to select 
a strategy, and correctly analyze the results of the move in order to 
store (remember) meaningful experience. In addition, it must form con- 


cepts regarding the reliability or honesty of its opponents. The parti- 


cipants in this game may or may not be completely truthful in their 
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negotiations, which is certainly characteristic of actual diplomats at 
the conference table. The computer, then, must form estimates of its 
opponent's reliability and factor this into its selection of a strategy. 
In the first game played, the computer has no prior knowledge upon 
which to draw, and so must reason from an analysis of the present situa- 
tion. With each move however, the system acquires more experience and 
thus has a better base from which to draw in selecting strategies. When 
the computer plays its second game, it has the experience of the first, 
with one winning and one losing strategy, to use for reference. In 
general, the more games it has played, the more experienced it is and 


the better it performs in making important decisions. 


- 
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III. THE GAME 


DIPLOMAT models the representatives of two nations interacting in 
strategic negotiations regarding armaments. It is basically a non-zero- 
sum two person rectangular game, using the phraseology of formal game 
theory. Each nation is given three BROT cett ot possible strategies re- 
garding armaments: 

Strategy 1: Increase Armaments (Arm) 

Strategy 2: Maintain the Status Quo 

Strategy 3: Decrease Armaments (Disarm) 
The following basic concepts govern the decision of the strategy to be 
followed by each side: - 

Each nation starts the game with zero strength and zero wealth, 
where "zero" implies a deviation from the average rather than absolute 
zero. 

Arming increases strength by one unit, disarming decreases strength 
by one unit, and maintaining the status quo does not change strength. 
Arming costs money decreasing wealth, disarming gains wealth, and main- 
taining the status quo may increase or decrease wealth, depending on the 
strength of the player: If the player is strong in armaments, it will 
cost him more for upkeep and maintenance and hence decrease wealth; if 
the player is weak, armament upkeep is low and maintaining the status 
quo should gain some wealth. The basic changes in wealth for the differ- 
ent strategies are shown as a function of strength of the player in 


Table I. 
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STRATEGY 


Strength 1 2 3 

S < -6 0 3 
~6 2S < -4 0 2 2 
eh 25 & -2 ~] 2 3 
~2eeG es +2 ~2 L 4 
+2) 5) = HN, ~3 0 3 
44<S £&= 46 o4 ~2 3 

S$ > +6 -6 ~3 2 


Table I 
Wealth Changes Versus Strategies As 
a Function of Player Strength 
Changes to the basic values in Table I are generated by a random 
integer amount between the values of -2 and +2. These changes, which 
are to simulate economic conditions, are generated at random times during 
the game; hence, a given economic condition may last for only one move 
or for many moves. The economic condition for each player may be differ- 
ent, as each opponent uses a different random number generated from the 
same random number seed. The economic cond:tion in effect for each 
player is not furnished to either opponent, but must be estimated based 
on the results of each move. The economic conditions of -2 and -1 mimic 
those times when prices aie high, and the conditions of +l and +2 mimic 
those times when the cost of living is relatively low. These values are 
added to the basic wealth change values of Table I to determine the 
actual wealth changes for the strategies chosen. 
Point values for each participant are determined after each move 
according to the formula: 


POINTS = (Previous) + 2 X (Wealth) + 5 X (Strength) 
Points Change Change 
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Bonus points are given for the following combinations of strategies: 

If both opponents cooperate in disarming, each receives two 
bonus points as a reward for their cooperation. 

If both opponents arm, each receives -1 bonus point, because 
both have spent money without acquiring any relative advantage. 

If one side arms and the other disarms, the opponent who arms 
is awarded four bonus points for "outfoxing" the other. 

The length of the game is at least ten and at most fifty moves. 
Between these values, a random selection is used for the decision to end 
the game; as the number of moves increases, the greater the chance of 
random termination. Experience has indicated that the average length of 
the game is twenty moves. 

The winner of the game is decided by one of three methods: 

Normal Termination: The participant with the most POINTS at 
the end of the game is declared the winner. 

Abnormal Termination: The game may be abnormally terminated, 
even before ten moves have been completed, in two ways. The game is 
stopped if one nation's wealth becomes thirty units greater than the 
other, and the richer nation is declared the winner. Similarly, if one 
participant becomes stronger in armaments than the other by ten units, 
the game is stopped and the stronger nation is declared the winner. 

In playing the game of DIPLOMAT, each side initially declares a 
proposed strategy called that participant's Concession Point, with each 
side taking turns declaring the first concession point. The proposal of 
each player is used as an aid in deciding that player's probable actual 


strategy. After both concession points have been declared, the computer's 
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move is locked into the system and the human participant is asked to 
declare his final strategy for that move. Of course, the concession 
point and the strategy need not be the same, but wisdom must be used in 
selecting proposals versus actual strategies in order to maintain a high 
degree of reliability, yet keep the opponent off guard as to the actual 
strategy to be chosen. 

The program is written as a main routine which controls the running 
of the game itself, and forty-one subroutines. Seven of the subroutines 
assist in controlling the game and in performing list-processing chores, 
nine of the subroutines assist the program in accomplishing its reasoning 
capabilities, and the other twenty-five subroutines are necessary for 
accomplishing the task of learning. Among the tasks performed by these 
forty-one subroutines are setting up the memory, saving and restoring 
experience, selecting a strate;sy and a concession peint, determining the 
opponent's reliability, pattern matching data from previous games and 
moves to take advantage of prior experience, and so forth. The program 
is written in the FORTRAN IV language, and is designed to operate on- 
Line on a computer terminal using the Cambridge Monitor System. A 
complete listing of the program is contained in Appendix C. 

Figure 1 is a macro-flowchart of the game. DIPLOMAT progresses in 
the following manner: 

After initializing counters, point values, and payoff tables, 
the system sets up the memory cells in an associative memory structure, 
filling in the data from previous games. The initial economic condi- 
tions are also determined. 

The player (hereafter called SOUTH) signs into the system 


with his name. The computer program (hereafter called NORTH) then 
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Figure 1. Macro Flowchart of the Game DIPLOMAT 
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checks to see if he has played this opponent before, and if so brings 
his history to the top of the catalog of players. 

The player inputs a random integer number for use as a seed 
in generating random numbers throughout the program. The computer simu- 
lates flipping a coin to see who will make the first proposal during the 
first move. SOUTH calls the flip of the coin. (In the discussion which 
follows, it will be assumed that SOUTH won the toss). 

Having won the toss of the coin, SOUTH must make the first 
concession point. To aid him in making his selection of a proposal and 
ultimately of a strategy, a payoff matrix is presented to him showing 
the approximate payoffs to SOUTH (relative to NORTH) for the various 
combinations of strategies. Approximate payoffs are shown rather than 
actual because neither SOUTH nor NORTH have access to the existing 
economic conditions. A sample payoff matrix as presented to SOUTH is 


shown in Figure 2. 


SOUTH 
STRATEGY Jniccidbideeicieiekeieeekeicisbictelek 
* 
N IL * ~] 1 2 
O * 
R 2 * -1 0 1 
at * 
H 3 * 2 ~] 2 


Figure 2 


Approximate Payoff Matrix Presented to 
Player (SOUTH) 


The approximate payoff matrix changes according to the strengths of 
NORTH and SOUTH because of the differences in wealth for the various 


strategies as shown in Table I. 
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SOUTH may propose a concession point of 1 (arming), 2 (main- 
taining the status quo), or 3 (disarming). NORTH then considers SOUTH's 
concession point, attempts to determine if SOUTH is being honest, analyzes 
his payoff matrix, and responds to SOUTH with his concession point. 

NORTH also decides upon a strategy at this time. 

SOUTH considers NORTH's concession point and decides upon a 
strategy for the move. Both sides then enter their strategies into the 
system and the results are tabulated and presented for analysis by both 
participants. 

For the next move, NORTH will declare his concession point 
first. The game proceeds in this manner, alternating between NORTH and 
SOUTH as to who is first to declare a concession point. After each move, 
each side analyzes the results in order to determine the economic condi- 
tions in effect. After ten moves have been completed and after each move 
thereafter, a random number is generated and tested to ARINC 1£ tne 
game should end. 

At the end of each move, changes to the payoff tables gener- 
ated as a result of changing strengths of the participants are calculated 
and inserted into the system, and it is determined if it is ae to 
change the economic conditions. If so, they are calculated aa inserted 
into the game. 

A sample output of the program as exhibited at the counter terminal 
is located in Appendix B. 

In playing this game, the computer program maintains data regarding 
past moves and past games in order to draw upon this experience in 


selecting strategies and concession points in future moves and in future 
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games. It thus forms concepts of each player as it proceeds. At the end 
of each move, some of the learning subroutines are called upon to update 
the short-term memory in order to store data for use in playing the game 
in progress and for maintaining running totals. At the end of the game, 
others of these subroutines calculate the game totals and determine the 
characteristics exhibited by both participants for inclusion in the long- 
term memory. This is the experience gained by the program from this game. 

The data which forms the experience is kept in an associative memory 
structure mainly for ease of manipulation, but it is considered that this 
patterns the human in organizing data and experiences in his mind. In 
Studies conducted of neural nets, it has been found that the human will 
take data, organize it into logical structures based on determining re- 
lationships between units of the data, and store it in an associative 
net accordingly [Ref.s 2 and 3]. 

Diagrams of the memory structure of DIPLOMAT are contained in Appen- 
dix A, and were conceived by oie authors after critical analysis of the 
structure of data maintained about players during hand simulation of the 
game. 

Besides exhibiting the human attribute of learning by storing away 
past experiences, the program mimics the human in its reasoning cap- 
ability in deciding upon strategies and concession points. The method 
of reasoning was also patterned after analysis of the mental reasoning 
used during hand simulation of the game. These two attributes of 
reasoning and learning are discussed in greater detail in the sections 


which follow. 
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A. THE REASONING PROCESSES 
Two of the subroutines of the program are designed to mimic the 
reasoning processes of the human. These are the strategy decision sub- 
routine (STRTGY) and the concession point decision subroutine (CPNDCN). 
These subroutines in turn call on many other subroutines to determine 
optimum strategies, probable moves of the opponent, next moves of the 
computer, and so forth. Both the strategy and concession point decision 
subroutines were written based on the thought processes used by the 
authors in playing the game by hand. 
ihe Subroutine STRTIGY 

This subroutine-has three main functions. The first is that 
of strategy decision based on an analysis of all the factors available. 
This decision is final if the player's concession point is known, other- 
wise it is a tentative decision until the declaration of SOUTH's con- 
cession point. The second function is that of reconsideration of the 
tentative strategy after SOUTH has declared his concession point, and 
results in the final selection of a strategy. This function is called 
upon only if NORTH was first to declare a concession point. The third 
function is analysis of the completed move in order to determine if a 
better choice of strategy could have been made. If so, an adjustment 
of the calculations performed in the decision portion of the routine is 
accomplished. 

Macro-flowcharts of STRTGY are shown in Figures 3 and 4. The 
subroutine may be called upon either two or three times each move, de- 
pending upon who submits the first proposal. A series of flags are 


used to determine for which function the subroutine is being called. 


The algorithm for making this determination is shown in Figure 3. 
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Figure 4 provides the basic algorithm of STRTGY. The decision 
upon a strategy for the move is based upon the consideration of four 
factors, each of which are weighted in a "polynomial'' fashion. These 
four terms of the polynomial are based upon: 

1)  SOUTH's concession point, 

2) POINTS relative to SOUTH, 

3) Previous experiencc, and 

4) Economic conditions. 
The coefficients assigned to each of these terms are inserted into the 
system at the start of the game, and may become modified as the game 
progresses, as discussed in the description of the analysis function. 

Even before considering any of the four factors, the computer 
checks to ensure that it is not in danger of losing because of relative 
wealth or strength disadvantage. If it is, the system immediately selects 
as its strategy the one which will gain the most wealth or strength, as 
needed, and returns to the main program without considering any of the 
four factors. If the computer finds that this emergency action is neces- 
sary, later analysis of the move is not peeforned: 

If emergency action is not necessary, each of the four factors 
are considered and used in determining a final strategy for the move. 

The first factor to be considered is SOUTH 's concession point. 
The computer first determines if it wants to call for a renegotiation. 
Renegotiation is requested if it finds that its opponent's proposal was 
to arm and that he is cuite strong already; it requires SOUTH to submit 
a new concession point, althoug!: the new proposal may be the same as 
the original. The next step after considering (and possibly carrying 
out) renegotiation is to attempt to determine SOUTH's probable forth- 


coming strategy. The computer selects candidates for its opponent's 
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strategy based on weighted reliability estimates, patterns of dishonesty 
in SOUTH's proposals, history of previous games with this player, and 
formal game theory. After deciding upon SOUTH's probable strategy, 

the computer selects as the first term of the polynomial the strategy 
which maximizes NORTH's gain relative to SOUTH if SOUTH actually selects 
that strategy. 

The second factor to be considered in selecting a strategy is 
based on NORTH's points relative to SOUTH. Depending upon whether the 
computer is behind, ahead, or even with its opponent, this term of the 
polynomial is set to the strategy which maximizes NORTH's possible gain 
Pr hinizes its possible loss. Formal game theory is used in selecting 
these possible strategies. 

The third factor to be considered is the most difficult one, 
because it is based upon previous experience. If SOUTH has been a pre- 
vious opponent (within the last ten opponents), there exists a history 
of his previous games in the computer's long term memory. NORTH can use 
this information in predicting SOUTH's probable moves. In addition, 
NORTH can search other strategy types contained in its libraries of past 
games in order to select previously successful strategies to use against 
its opponent. After several moves have been completed, the computer 
searches all the strategy listings in its “experience” in order to classify 
SOUTH's general pattern of strategies. Each of these listings may con- 
tain pointers to other stratcgy patterns which have proven successful 
against SOUTH's pxttern in the past. NORTH can also pattern-match its 
own strategy pattern against those existing in the library in order to 


determine its own predicted move. The computer occasionally selects a 
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strategy opposite to the predicted one to avoid being stereotyped by 
SOUTH. This is called a "guess-opposite" selection of a strategy. It 

is the third factor of the polynomial mien gains from the learning 
capabilities of the machine. The method of learning and pattern-matching 
is discussed in greater detail in a later section. 

The fourth and final factor to be considered in selecting a 
strategy is based on an estimate of the economic condition in effect. 
After the completion of each move, the program analyzes the results to 
determine if they match the expected values. If not, it estimates the 
economic conditions and sets this term of the polynomial to the strategy 
which takes the most advantage of the state of the economy. For example, 
if prices are low, it is. probably the best time to arm, but if prices are 
abnormally high it may be too expensive to arm at that time. 

After all four factors have been considered in selecting a 
strategy, the system conducts a vote to determine the choice for the move. 
If SOUTH's concession point has not been declared, the coefficient of the 
first term is set to Zero, and a tentative strategy is chosen based on 
the other three terms. When it becomes time for the reconsideration, 
the vote is taken of all four terms for deciding the final strategy. 

The vote is accomplished by summing the coefficients of the terms voting 
for each of the three possible strategies. The strategy which receives 
the greatest sum is the "winning" strategy; in case of a tie, the com- 
puter selects from the tieing strategies the one which will result in 
the greatest absolute point gain. 

Having selected a strategy, control is returned to the main 
program which either calls upon Subroutine CPNDCN to decide NORTH's con- 
cession point, or else locks the strategy into the system awaiting SOUTH's 


indication of a strategy. 
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The third function of Subroutine STRTGY is that of analyzing 
the completed move in order to determine if a better strategy could have 
been chosen. The system does this by entering NORTH's relative payoff 
table with the actual strategy chosen by SOUTH to determine the strategy 
which provides the maximum relative gain. If the computer determines 
that the correct strategy was chosen, no adjustments are performed and 
‘control is returned to the main program. If, however, the best strategy 
was not picked by NORTH, the computer determines if any of the votes cast 
for the different strategies matched the best possible strategy. If any 
are found, the coefficients corresponding to those terms are then in- 
creased (tuned up), and those assigned to the wrong terms tuned down. 

If no votes are found, the analysis was unsuccessful for that move. 

If the conputer ultimately wins the game, the final values of 
the coefficicnts for each of the terms are inserted into the long-term 
memory of the player for use as the initial values in the succeeding game 
with that player. This is done because the system found these coeffi- 
cients successful and they would probably provide a better base from 
which to start the next time. This is but another part of the learning 
process of the system. 

Thus, Subroutine STRTGY attempts to determine a strategy to be 
followed in the game of DIPLOMAT in much the same way that a human reasons 
through the game. Similar to a human, if the reasoning process results in 
a wrong answer, the system attempts to improve itself. 

ae Subroutine CPNDCN 

This is the second of the program's subroutines which employs 
human-like reasoning. Its purpose is to determine a concession point to 
be proposed to SOUTH after having selected a tentative (or final) strategy. 


Figure 5 provides a macro-flowchart of this subroutine. 
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NORTH's concession point must be carefully chosen. In order 
to keep the player off guard, the computer must try to maintain at least 
a facade of reliability, yet not be so reliable as to be "read like a 


book." 


If NORTH and SOUTH are each following a pattern of moves, the 
computer should indicate that it expects Co follow the pattern, even 
if it has no such intention. 

For the initial moves, the computer uses a semi-random pattern 
of selecting proposals. Weight is given to choosing either the actual 
strategy proposed, or to choosing a passive (i.e., disarming or status 
quo) declaration. After the initial moves have been completed, however, 
the computer utilizes much of the data it has been accumulating on the 
player during the game. In addition to pattern-matching previous moves, 
the computer maintains estimates of the player's reliability as well as 
its own. EN is NORTH's estimate of SOUTH's honesty, and ES is what NORTH 
thinks SOUTH estimates for the computer's reliability. These are both 
weighted values, with the most weight given to the recent moves. EN and 
ES are calculated after each move by comparing the actual strategy with 
the concession point. The values of EN and ES drop accordingly each time 
there is not a match. 

If ES is high, is increasing, or at least is greater than EN, 
the system determines its predicted move from pattern-matching its own 
strategies, and uses a predicted move as its concession point. If, on 
the other hand, NORTH's reliability needs improving, the subroutine 
normally selects the actual strategy determined by STRTGY as the computer's 
concession point. 

As with STRIGY, CPNDCN was written based on hand simulation of 


the game. Notes were kept on the reasons behind each decision, and 
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used to help determine the methods of reasoning developed in these 


subroutines. 


B. MEMORY STRUCTURE AND THE LEARNING PROCESS 

The information within this section is somewhat more detailed be- 
cause it describes the attempts to exemplify the concepts of Piaget in 
structuring the mind. Computer list-processing techniques are used in 
Structuring and manipulating the memory for this task. The reason for 
this is that these techniques model the theoretical structures of the 
brain and are efficient in coding requirements. 

is The Memory Structure 

The memory structure of the program, depicted in Appendix A, 
is divided into two segments, the "temporary" or short-term memory and 
the ‘permanent:’ or long-term memory. The short-term memory contains 
data regarding the current game and the long-term memory contains the 
data which comprises the experience gained from previous games. 

The short-term Re OEy is used to maintain data necessary for 
determining the pattern of play being used by both the human player and 
the computer. The information within this segment of memory grows as 
the game progresses. At the end of the game, pertinent data is summarized 
and transferred to the long-term memory, and the ator cen memory is 


deleted or "forgotten." 


The temporary memory consists of arrays and 

lists for determining the reliability estimates used in the reasoning 
processes of the syst’n. In addition, this segment of memory contains 
a pattern of strateg es used by each participant. It also maintains a 


total of the number of times each strategy is selected. The former is 


used in generally classifying the method of play which is being employed, 
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and the latter is used in determining the overall aggressiveness being 
exhibited. 


The long-term memory, the "experience," 


consists of arrays and 
lists arranged into four libraries. Libraries are interconnected as 
they are developed, in order to maintain continuity in the flow of in- 
formation and facilitate access to specific data. These four libraries 
are labeled the "Catalogue of Players," the "Library of Initial Moves," 


' and the "Library of Sequences of Moves." 


the "Library of Types of Moves,' 
For simplicity these names are abbreviated the "Catalogue," the 'Initial 
Library,'' the "Type Library," and the "Sequence Library." 

To provide a basic understanding of the construction and main- 
tenance of these libraries, it is necessary to define various terms associ- 
ated with them. A move is a played strategy. Groups of three moves are 
categorized into five levels of aggressiveness. Groups are thus combina- 
tions of By three strategies played. The following example illustrates 
the construction of groups: 

Suppose an opponent picked the following moves (strategies): 

owen tl  S 5S | Lol» 2 1 3 
Group (1) then consists of 1 2 1, Group (2) is 2 1 1, Group (3) is 
1 13, Group (4) is 1 3 2, and so on. | 

Patterns are determined in the Initial Library according to 
individual moves, and are determined in the Type Library by a sequence 
of groups. The sequence of groups is called a ''TYPE."' Further dis- 
cussion will be given to the utilization of categories of aggressive- 


ness, and their associated groups, in the explanation of the learning 


process of the program. 
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The Catalogue contains information on one to ten players. As 
a player is first introduced into the game, a unique set of nodes (sequen- 
tial set of computer words) is established for him which at the end of 
the game will be filled with data and pointers regarding his history of 
play. The most recent player is maintained at the top of the catalogue 
and the oldest player at the bottom. As players participate in addi- 
tional games, a reordering of the Catalogue is accomplished to maintain 
this perspective. If the Catalogue reaches its capacity of ten players 
and a new player is introduced into the system, the information on the 
oldest player is deleted (forgotten) and the new player established in 
the Catalogue. ; 

Information maintained on each player includes pointers to the 
initial strategies and a sequence of "TYPE" patterns used by both the 
player and the computer in previous games. It also contains the set of 
parameters (the foundation of the coefficients considered in fe strategy 
polynomial) which have proven most successful against that playe:, the 
average values of reliability demonstrated by both the player and the 
computer when playing against that player, and the player's aggressive- 
ness in previous games. | 

The INIT Library contains the initial three moves (first group) 
used in various games in the past, and pointers to the Type Library. 
Information on the exact initial strategies used in previous games is 
maintained so that the computer can try to get a "jump" on the player at 
the beginning of the game. It is used to predict SOUTH's initial strategy 


and to aid the computer in selecting the best offense against this 


strategy. 
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fine Type Library contains up to ten structures. A structure 
consists of a header cell followed by a TYPE of eight to ten groups. 

The header cell contains pointers to other structures in the Type 
Library which have demonstrated a successful defense against this TYPE, 
and against which this TYPE has proven to be a good offensc. 

The Sequence Library is a history of the TYPEs and associated 
initial strategies for each game played by a particular player in the 
Catal@gue. It also contains the computer's TYPEs used against any player 
in the Catalogue. Each entry in the Sequence Library contains a pointer 
to a structure in the Type Library and a pointer to the initial strategy 
associated with that structure. 

When the capacity of each of these libraries is filled, the 
oldest information is deleted, or ':pdated, to make room for the new in- 
formation. This conforms to the contention that the oath grows to a 
finite size, then information becomes restructured or forgotten as it 
grows out of date. 

2. The Learning Process 

The learning process progresses as the memory structures 
develop. During the first game the computer plays, there is nothing in 
long-term memory so the computer must reason through the game as best 
it can. During succeeding games, however, the computer searches long- 
term manieey for strategy predictions and pattern-matching types of 
strategies. 

When the player signs into the system at the start of a game, 
the computer searches the Catalogue to determine if this player has 


been played before. If so, the computer assumes that he will follow 
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the same initial strategy and determines the best moves against that 
strategy. After two moves the computer pattern-matches these against 
the INIT Library to find the closest match. The computer's third move 
is selected as the one which is best against the third move listed in 
the closest matched INIT group. After the third move, the computer 
again pattern-matches the initial three moves against the INIT Library, 
finds the closest match, and predicts that the player will use the TYPE 
to which that entry in the INIT Library points. 

The Type Library provides general categories of strategies to 
simplify pattern-matching. In a twenty-move game, there could be an 
almost infinite number of sequences of exact strategies used, but by 
categorizing exact strategies into general classifications, pattern-match- 
ing may be done at a meta-level. These categories, as stated earlier, 
are determined by arranging the moves into groups of three and classi- 
fying the group into one of five levels of aggressiveness. Figures 6 
and 7 illustrate the techniques of categorizing strategies and making 
predictions based on the types obtained. The overlap of the groups pro- 
vides continuity in classifying and predicting. 

As the game progresses, the computer pattern-matches the TYPEs 
used by both the player and the computer, which are maintained in short- 
term memory, against the Type Library. This provides NORTH with pre- 
dicted moves for both participants, and also with an indication of the 
TYPE that is best to use against SOUTH's TYPE. 

The Sequenc:: Library provides the computer with the ability 
to make better predictions of the TYPEs that a player will use. After 


each game with a player, a pointer to the TYPE employed in that game is 
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The sum of the strategies for each category is: 


Category 1 - 3 or 4 


Category 2 - 6 


Figure 6. 
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By totaling the first two moves of any predicted category, it is 


possible to predict the next move: 





it Parse For The Move 
Two Moves Category Predicted 
Total: Number: Is: 
Z l 2 
3 1 1 
2 Al 3 
3 Al 2 
Overlapping of 4 Al 1 
Category Groups 
Shown Above 3 2 3 
4 Z 2 
Example Strategies 5 2 1 
and Categories are . 
Inserted 4 A2 3 
5 A2 Ja 
6 A2 l 
5 3 3 
6 3 2 


Figure 7. Prediction of the Next Move Based on 
Strategy Type 
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inserted in the Sequence Library. The computer can pattern-match these 
lists to determine the expected type which the player will use during 
the next game. 

At the end of a game, corresponding data between the short- 
term memory and the long-term memory are compared. This is accomplished 
by the same methods of pattern-matching used during the game. If similar- 
ity exists, the structures of the short-term memory are combined with the 
closest Initial and Type Library structures to form more up-to-date in- 
formation. If no similarities exist, the information from temporary 
memory is transferred to permanent memory as new Initial and Type Library 
entries. If these libraries Are at capacity, the oldest information is 
deleted (forgotten) and the new information is inserted. 

As suggested by the theory, past experience is utilized to 
obtain the best prediction of events, but if the computer finds no match- 
ing experience upon which to draw, it must reason through the problem as 
best it can. Knowledge grows when new structures are formed or old ones 


reconstructed. 


C. ANALYSIS OF THE GAME 
After a rather poor start, the computer has gone ahead of its op- 


This 





pa 


ponents in total points for all games, and the gap is wide 
is due partly to some minor changes in the strategy decision routines, 
however evidence indicates that most of the credit can be given to the 
building of the libraries. This contention is supported by the fact 

that the analysis portion of the strategy routine tends to increase 

(tune up) the basis of the coefficient assigned to the experience factor, 


and tune down some of the others. 
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Based on observation of the game and discussions with those playing 
it, it is evident that the players and the computer use many of the same 
factors in deciding upon a strategy and a concession point. In addition, 
concepts built by the human are similar to the concepts formed by the 
computer and stored within its memory. The average values for the re- 
liability estimates that the computer maintains on both itself and its 
opponent remain relatively constant as the game progresses. The same is 


true of the weighted reliability. 
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IV. CONCLUSIONS 


"If a program can be constructed that, when placed 

in a prisoner's dilemma situation, exhibits behavior 

like the behavior of people when placed in the same 

Situation, then that program will be a powerful tool 

for generalizations." 

The above statement by Professor Weil is repeated from an earlier 
section of this thesis for emphasis. In its existing form, this program 
does provide a powerful tool for extracting generalizations regarding 
human behavior in a medium risk decision making task. With but minor 
changes in wording or by incorporating extensions to the game, the pro- 
gram can be made applicable to almost any field of corporate or govern- 
mental endeavor requiring a psychological understanding of human be- 


havior in making decisions where different gains can be achieved at 


varying risks. 


A. EXTENSIONS AND CHANGES 

DIPLOMAT is in itself an extension to the classic prisoner's 
dilemma game. It may be extended or changed to broaden its applicabil- 
ity to real world situations and make the game considerably nore inter- 
esting. 

The simplest change to the program would be to change the name 
assigned to the three strategies. For example, the applicability of 
the program could be changed by transforming the words arm/status-quo/ 
disarm into the words buy/wait/sell, or perhaps raise-prices/no-change/ 
lower-prices. 


The whole outcome of the game can be changed drastically by changing 


46 





the POINTS formula. This is done by simply assigning different co- 
efficients to the factors of WEALTH and STRENGTH, or by changing the 
values in Table I (the wealth changes for the various strategies shown 
as a function of player strength). Thus the game may be altered to 
match actual conditions encountered in, say, the business community. 

The number of strategies from which to choose could be changed in 
either direction. Decreasing the choice to two would render the game 
closer to the prisoner's dilemma situation, but even this has many ap- 
plications in the real world. On the other hand, it would be more in- 
teresting to increase the number of strategies. For example, there 
could be two levels of arming and two levels of disarming, or for buying 
or selling. Some implications of increasing the number of strategies, 
however, would be that the pattern-matching routines may not be feasible 
in their present state. It would probably require a pattern-matching 
scheme which placed more emphasis on the meta-level, i.e., looking at the 
broad spectrum of the pattern roe a higher level, rather than pattern- 
matching individual strategies or small groups of strategies. 

Increasing the number of players is probably the most difficult of 
the possible extensions to the game, but provides the most interesting 
possibilities. If the number of players is Wcee nese treaties peeueee 
the players and alliances among groups of players may be proposed and 
formed. An infinite number of situations may arise out of Ge idea, for 
example, one nation might wonder whether his ally will abide by the al- 
liance or possibly turn on him several moves hence; or, a player might 
hesitate to sign an agreement when a better one might be offered from a 
different player. In the diplomacy situation, several nations may dis- 


arm to gain wealth, then form an alliance against a stronger nation which 
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has armed to gain strength at the cost of wealth. Increasing the number 
of players would also make the concepts of reasoning and learning more 
difficult, but more fascinating. For example, will one nation risk an 
alliance with another when it remembers prior treachery? Instead of 
merely analyzing what one's opponent thinks of him, a player will have 
to analyze several players' estimates of all the opponents. The reli- 


ability considerations become almost overwhelming. 


B. POSSIBLE USES OF DIPLOMAT 

It is considered that Professor Weil was correct in assuming that 
the prisoner's dilemma computer game would be a powerful tool for 
generalizations. The Paine of DIPLOMAT or its extensions would be an in- 
valuable aid in the training of executives prior to stepping into posi- 
tions requiring the art and finesse of personal contact. The speed of 
the computer permits the game to proceed rapidly and permits many differ- 
ent situations to be established by changing payoffs anc formulas. The 
performance of different players against standard setups could then be 
analyzed. 

In addition, personnel in the study of behavioral science and psy- 
chology could develop a better understanding of the nature of the human 
decision-making process and of the risk of striking out on an independent 
path as opposed to the benefits and security gained by cooperating. The 
fact that the risky path may lead to greater gains may be more important 
to some people than to others. 

The training of college students in the theory of marketing and 
analysis would also be enhanced by applying textbook concepts to the 


difficulties of the everchanging conditions and types of people with 
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which they may deal. This training might prove useful for young officers 
in the diplomatic corps or even at junior level armed forces colleges. 

In order to enhance the development of computer application to this 
type of work, it is recommended that consideration be piven to the 
possibility of adapting this program to marketing or financial decision- 
making courses in the managem-nt curriculum. This type of program could 
be extremely useful as a tool in teaching management students the power 
and usefulness of the computer. It demonstrates the interface capabil- 
ities of man and machine. It can also be used as a teaching aid in 
artificial intelligence, ge:re theory, and basic management-decision 
courses. The first part of Section III describing the game could be 
reproduced for use as a handout for potential players. 

The concept and performance of DIPLOMAT appears to be good. It is 
ahead of the human players in POINTS and performs well in adapting to 
different human strategies that have been tried against it. However, 
Similar to a human, it does not win every game, indicating that even it 


has more to learn. 
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APPENDIX A 


THE MEMORY STRUCTURE OF DIPLOMAT 
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